@arXiv_csCL_bot@mastoxiv.page2024-05-01 06:48:59
Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom
Shisen Yue, Siyuan Song, Xinyuan Cheng, Hai Hu
https://arxiv.org/abs/2404.19509 https://arxiv.org/pdf/2404.19509
arXiv:2404.19509v1 Announce Type: new
Abstract: Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs' performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.
@servelan@newsie.social2024-02-14 03:06:04
Trump may try to whitewash his racism by picking Tim Scott as VP, but he's fooling no one
https://www.dailykos.com/stories/2024/2/13
@spamless@mastodon.social2024-03-24 14:34:03
First gene-edited pig kidney transplanted to a living human - The Washington Post
#medicine
@frostpoem@mastodon.art2024-02-17 07:18:51
That delicious feeling when an app you've feared for a long time might be dying pops up with an update, some new content and what looks like an active roadmap for the time ahead. Directly related, NatureSpace is truly like no other for slipping into some of the most vivid soundscapes I've ever seen--and not all are for sleeping to! They've just added a subscription service, but you can also purchase sounds as you like them/in packages, which I think is a really nice way of handling that.
@rberger@hachyderm.io2024-02-15 20:38:39
“When it comes to big troubles, the Republican counties suffer far and away the worst problems – and they’re getting even worse. Unlike fraudulent, “cherry picked” political claims that select only certain years to show whatever trend partisans want, I use mathematical trendlines that incorporate all years in the period into a single trend in the following charts.”
#TrumpPoxOnSociety…
@servelan@newsie.social2024-03-18 15:28:21
👍 Jack Smith shreds Reagan AG Ed Meese's amicus brief's claim that Smith is an illegal prosecutor
https://www.dailykos.com/stories/2024/3/17
@rberger@hachyderm.io2024-02-15 20:38:39
“When it comes to big troubles, the Republican counties suffer far and away the worst problems – and they’re getting even worse. Unlike fraudulent, “cherry picked” political claims that select only certain years to show whatever trend partisans want, I use mathematical trendlines that incorporate all years in the period into a single trend in the following charts.”
#TrumpPoxOnSociety…
@arXiv_csHC_bot@mastoxiv.page2024-03-18 06:49:55
Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Tracking
Yiyang Gu, Yougen Zhou, Qin Chen, Ningning Zhou, Jie Zhou, Aimin Zhou, Liang He
https://arxiv.org/abs/2403.09717
@arXiv_csHC_bot@mastoxiv.page2024-04-05 07:26:54
Toward Safe Evolution of Artificial Intelligence (AI) based Conversational Agents to Support Adolescent Mental and Sexual Health Knowledge Discovery
Jinkyung Park, Vivek Singh, Pamela Wisniewski
https://arxiv.org/abs/2404.03023
@arXiv_csHC_bot@mastoxiv.page2024-04-05 07:26:54
Toward Safe Evolution of Artificial Intelligence (AI) based Conversational Agents to Support Adolescent Mental and Sexual Health Knowledge Discovery
Jinkyung Park, Vivek Singh, Pamela Wisniewski
https://arxiv.org/abs/2404.03023